Matrix Eigen-decomposition via Doubly Stochastic Riemannian Optimization: Supplementary Material

Preparation First, based on the definitions of A t , Y t , ˜ Z t and Z t , we can write g t = G(s t , r t , X t) = p −1 st p −1 rt (I − X t X ⊤ t)(E st ⊙ A)(E ·rt ⊙ X) = (I − X t X ⊤ t)A t Y t. Then from (6), we have X t+1 = X t + α t g t W t − α 2 t 2 X t g ⊤ t g t W t. Since W t = (I + α 2 t 4 g ⊤ t g t) −1 = I − α 2 t 4 g ⊤ t g t + O(α 4 t), we get X t+1 = X t + α t A t Y t − α t X t X ⊤ t A t Y t − α 2 t 2 X t g ⊤ t g t − O(α 3 t). Let F t be the set of all the random variables seen thus far 1 (i.e., from 0 to t). Proof. The proof technique follows (Balsubramani et al., 2013) and (Xie et al., 2015). Note that for two square matrices Q i , i = 1, 2, their products Q 1 Q 2 and Q 2 Q 1 have the same spectrum. The spectral norm (i.e., matrix 2-norm) is orthogonal invariant. Hence, we can write λ min (Z ⊤ t VV ⊤ Z t) = λ min (V ⊤ Z t Z ⊤ t V) = min y̸ =0 ∥V ⊤ Z t y∥ 2 2 ∥y∥ 2 2 = min y̸ =0 ∥V ⊤ Z t y∥ 2 2 ∥Z t y∥ 2 2 = min y̸ =0 ∥V ⊤ (X t + α t A t Y t)y∥ 2 2 ∥(X t + α t A t Y t)y∥ 2 2. 1 Mathematically, it's known as a filtration, i.e., sub-sigma algebras such that Ft ⊂ Ft+1.